Application directed user interface remoting using video encoding techniques

ABSTRACT

Methods, techniques, and systems for user interface remoting using video streaming techniques are provided. Example embodiments provide User Interface Remoting and Optimization System (“UIROS”), which enables the efficient remoting of user interfaces on behalf of their guests using generic video streaming techniques, such as H.264, to send compressed user interface image information in the form of video frame encoded bitstreams. With application cooperation, pixels are explicitly cached on the server using lookahead methods to lower latency in remoting the user interface for certain operations. In one embodiment, the UIROS comprises server side support including a UI remoting server, a video encoder, and rendering support and client side support including a UI remoting client, a video decoder, and a display. These components cooperate to implement optimized UI remoting that is bandwidth efficient, low latency and CPU efficient.

TECHNICAL FIELD

The present disclosure relates to methods, techniques, and systems foruser interface remoting and, in particular, to methods, techniques, andsystems for application directed, efficient remoting of interactive userinterfaces using video streaming and/or video encoding technologies.

BACKGROUND

The computing industry is evolving to a model of computing where thebulk of storage and computing occur at a datacenter or in the “cloud”(e.g., networked, Internet-accessible storage) and rich user experiencescan be remoted to the user's location, using client devices of manydifferent form factors. Significant advancements in virtualizationinfrastructure, networking infrastructure, as well as the diversity andproliferation of highly capable and portable client devices, have madesuch remote access highly viable and desirable. For example, it is notuncommon for employees, especially of large organizations, to workremotely and still desire and/or require use of their desktops andapplications at their home offices. This has become possible throughvirtualization technology that allows a user's desktop and applicationsto be run in a datacenter while the actual user interface (theinput/output—“I/O”—to the desktop and applications) is mimicked on aremote client device. Mimicking of the user interface is accomplished by“remoting” the user interface—that is directing (e.g., sending,forwarding, transmitting, communicating, or the like) screen output tothe remote device (for example, to a display associated with the clientdevice) and receiving input device data from the remote device (forexample, through keyboard, mouse, touch, or other device input). Theentire desktop (for example, the user interface of an operating systemrunning on a host computing system or virtualization server) or a singleapplication running on the desktop may be remoted on devices such assmart phones, tablets, notebook personal computers (PCs), desktop PCs,smart TVs, other form-factor computing devices, and the like.

Some challenges, including the latency of data arriving to and from suchclient devices, which impact the user experience, remain. Latency may bea result of limitations of the client devices and/or the networksconnecting the client devices to the servers where the desktops and/orapplications are running (or hosted). In addition, some networks and/orclient devices have bandwidth limitations that can make it difficult topresent a rich user interface experience in a responsive way. The moredata that needs to be transferred quickly to a client device, the morelikely latency and bandwidth limitations are encountered.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example virtualization server computingsystem for executing example embodiments of enhanced user interfaceremoting.

FIG. 2 is an example block diagram of components of an example UserInterface Remoting and Optimization System.

FIG. 3 is a block diagram of an overview of example video streamingusing an example generic video streaming protocol.

FIG. 4 depicts an overview of an example video encoding techniqueperformed according to the H.264 video streaming protocol.

FIG. 5 depicts an example of pixel movement according to an exampleoperation of a user interface.

FIG. 6 is a flow diagram of example server logic for enhanced userinterface remoting according to an example embodiment.

FIG. 7 is a flow diagram of example logic for analyzing a user interfacefor determining reusable pixel data according to an example embodiment.

FIG. 8 is a flow diagram of example logic for determining and generatingan encoding of user interface data using a video streaming protocolaccording to an example embodiment.

FIG. 9 is a flow diagram of example client logic for decoding andrendering a received video bitstream according to an example embodiment.

DETAILED DESCRIPTION

Embodiments described herein provide enhanced computer- andnetwork-based methods, techniques, and systems for user interfaceremoting using video streaming techniques. As used herein, the phrase,“user interface remoting” refers to either remoting an entire desktopand/or an application user interface unless noted otherwise. Exampleembodiments provide a User Interface Remoting and Optimization System(“UIROS”), which enables virtualization environments to efficientlyremote user interfaces on behalf of their guests (guest operatingsystems and/or guest applications) using generic video streamingtechniques, such as the well-known H.264 video encoding standard, tosend compressed user interface image information in the form of videoframe encoded bitstreams. A generic video stream technique (e.g.,format, codec, or the like) refers broadly to one not specificallyoptimized or designed for user interface remoting and thus may includeformats that can be decoded, for example, by off-the-shelf, proprietary,standardized, or other video decoders (e.g., generic video decoders).This enables a client device with a generic (often hardware or embedded)video decoder to perform user interface (UI) remoting without beingspecifically or specially programmed to do so.

The UIROS exploits user interface semantics to take advantage of thehigh “frame” coherency that user interfaces commonly exhibit. That is,for many user interface operations, only a small portion of pixelschange between consecutive frames (frames as defined by the videostreaming protocol used). Commonly, for example in user activities suchas scrolling, window moves, uncovering regions, and the like, a set ofpixels which have already been transferred to a client device arere-positioned (e.g., moved to a new location) in the target frame, butnot changed otherwise. This allows client devices to cache and reusepixel information on the client side receiving only information relatingto how the pixel information has moved in the target frame and any newinformation that needs to be rendered. Example embodiments of the UIROSadvantageously send only these updates as video frame encodedbitstreams, so that client devices receive more compact and fasterupdates thereby increasing the overall efficiency of remoting userinterfaces and thus decreasing UI latency. Client devices may includeany type of device that supports the video streaming protocol employed,such as for example smart phones, tablets, notebook computers, desktopcomputers, smart televisions (TVs), and the like.

In addition, by using generic video streaming protocols, such as H.264(otherwise known as Advanced Video Coding), H.265, Microsoft's VC-1, orGoogle's VP8, the UIROS can exchange optimized user interface remotingdata with client devices without need for a specialized, potentiallyhigh cost, proprietary device driver or specialized hardware installedon the client. This is because many client devices are shipped alreadywith hardware accelerators that support video decoding using thesestandards, and most popular web browsers (such as Internet Explorer,Google, and Chrome) are now configured (or in the near future will beconfigured) to use HTML 5, which supports a <video> tag for providing astandard network protocol for transmitting a video stream, such as H.264over TCP/IP, which may be passed through to a resident hardwareaccelerated decoder, if available. The H.264 standard, described furtherbelow, defines the syntax of an encoded video bitstream in addition tothe techniques for decoding the bitstream. Although described withreference to H.264, it is to be understood that the techniques, methods,and systems described herein also will operate with any other genericvideo standard, and that H.264, VC 1, and VP8 are simply examplesthereof. Note that the techniques, methods, and systems described hereinwill operate with proprietary as well as standardized or public videoformats (for example, those approved by a standardization organizationor publicly available); however, one of the desired outcomes of thesetechniques is to operate with standard web browsers executing onunmodified client devices that include generic video decoders in orderto make optimized user interface remoting viable for everyone on allsorts of client devices. Using a generic protocol and directing updatesto a web browser (or the native video driver) that provides built-insupport for video streaming in client devices eliminates a need tocreate specialized client devices with specialty device support for UIremoting.

In typical video streaming, different techniques, including exhaustivesearching, are used to determine motion estimation—to determine howpixels change from frame to frame. Motion estimation is the process offinding similar blocks of pixels (of a defined size, for example 16×16or 4×4) to a target block of pixels. Similarity is defined in differentways by different algorithms. If a similar block of pixels is found,then only movement and/or difference data needs to be transmitted (e.g.,communicated, forwarded, sent, or the like) and a reference to thesimilar block of pixels instead of all of the pixels. Exhaustivesearching techniques for determining motion estimation are CPU intensiveand sometimes prohibitive.

As described in detail below, using video streaming techniques for UIremoting presents opportunities for optimization that do not use suchexhaustive searching techniques. Because of the frame coherency of manyuser interface activities, the UIROS can more easily locate similarframes during the motion estimation process. For example, the UIROS maydetermine pixel reusability by examining one or more of the following:the frame buffer used by the guest in addition to analyzing user inputdevice movement, the graphics commands sent by the guest to thevirtualization support code (such as the virtualization logic, or acomponent thereof) or to the host computing system, and commands invokedby a guest application using an applications programming interface(“API”) directed to this purpose as described further below. Thisexamination process may provide a “fingerprint” of activities thatresult in or should result in reusable blocks. The UIROS may alsoanalyze feedback information about client load and network conditions toadjust (e.g., calibrate, tune, etc.) the timing and/or content of thevideo encodings and streaming.

FIG. 1 is a block diagram of an example virtualization server computingsystem for executing example embodiments of enhanced user interfaceremoting. Virtualization server computing system 100 may be configuredon a personal computer, a laptop, or server (host) hardware platform101, such as an x86 architecture platform. Note that a general purposeor a special purpose computing system suitably instructed may be used toimplement the virtualization server computing system 100. Thevirtualization server computing system 100 may comprise one or moreserver computing systems and may span distributed locations. Inaddition, each block shown may represent one or more such blocks asappropriate to a specific embodiment or may be combined with otherblocks.

In the embodiment illustrated, host hardware platform 101 may comprise acomputer memory 102, one or more central processing units (“CPU”) 103, aframe buffer (“FB”) 104, and one or more network connections, accessiblefor example via network interface card (“NIC”) 105. In addition, thehost hardware platform 101 may optionally comprise other components suchas one or more displays 109, graphics processing units (“GPU”) 108,input/output (“I/O”) devices 111 (e.g., keyboard, mouse, CRT or LCDdisplay, etc.), or other computer readable media 110.

Virtualization logic 120 is loaded into memory 102 of host hardwareplatform 101 and may execute on one or more CPUs 103. Virtualizationlogic 120 may alternatively be implemented in software, hardware, orfirmware, or some combination thereof. Virtualization logic 120,includes one or more virtual machine monitors (VMM) 142 a-142 c and VMXprocesses 151 a-151 c, which can support multiple virtual machines (VM)141 a-141 c, which can concurrently be instantiated and executed. Asused herein a “virtual machine” or VM is an abstraction representing theexecution space that a guest operating system and applications (the“guest”) may execute within, such as VM 141 a-141 c. Each virtualmachine 141 a-141 c may include a guest operating system (guest OS),e.g., guest OSes 143 a-143 c, and one or more correspondingapplications, e.g., guest applications 144 a-144 c, running on eachrespective guest OSes 143 a-143 c. In one example embodiment, each VM,when executing, is made accessible to a different user who is remotelyconnected from a different client connection. The number of VMssupported by any particular host may vary, for example, based on aspectssuch as the hardware capabilities, virtualization logic configurations,and desired performance. Other code 161 may also execute onvirtualization logic 120.

Each VM 141 a-141 c may require virtualization of one or more aspectsimplemented by the virtualization logic 120 and/or the host hardwareplatform 101. That is, the virtualization logic 120 may provide emulatedhardware and drivers to each VM. For example, through the VMX processes151 a-151 c and the VMMs 142 a-142 c, the virtualization logic 120 mayprovide one or more of a virtual CPU (“VCPU”), a virtual memory(“VMEM”), virtual device drivers (“VDD”), a virtual file system andvirtual disks, virtual network capabilities, and virtual graphicscapabilities, such as virtual graphics adaptors drivers and commandemulation, and the like. Each virtualization environment may function asan equivalent of a standard x86 hardware architecture platform such thatany operating system (e.g., Microsoft Windows®, Linux®, Solaris®86,NetWare, FreeBSD, etc.) may be installed as a guest OS (e.g., guest OS143 a-143 c) to execute applications in an instantiated virtual machine.Note that in other embodiments, virtualization of other hardwarearchitectures may be supported.

In one embodiment, the virtualization logic 120 provides virtualizedstorage support through a distributed VM file system 132, storage stack131, and device drivers 130 that communicate with the physical datadrives 106 and 107. In addition, the virtualization logic 120 providesvirtualized network support through a virtual switch 133 and networkstack 134 to communicate with NIC 105 of the host hardware platform 101.This support may be used to provide the TCP/IP connections at thevirtualization logic level referred to elsewhere herein. Also, thevirtualization logic 120 provides virtualized graphics support throughthe SVGA (or VGA) graphics adaptor implementations which use the servergraphics API 121 (such as OpenGl, Xserver implementations, etc.) tocommunicate with graphics drivers 122 that manage and fill frame buffer104 of the host hardware 101 using graphics commands. In certainembodiments, the graphics capabilities of the host hardware platform 101may be accelerated through the use of one or more GPUs 108.

In some embodiments, the virtualization execution environments areprovided through both a process executing at USER (less privilegedmode), referred to as the VMX process (e.g., VMX processes 151 a-151 c)and the VMM executing in a more privileged state (e.g., VMMs 142 a-142c). Each VM 141 a-141 c effectively executes in the process space of itsrespective VMX process 151 a-151 c (that is its memory is mapped to eachrespective VMX process). A VMX process, for example processes 151 a-151c, may comprise an MKS (mouse, keyboard, screen) thread (e.g., thread152 a) for processing input and output from the respective VM, e.g., VMs141 a-141 c. In one example UIROS, this is where the UI remoting logicand support resides and executes, as will be described in detail below.A VMX process also includes USER mode graphics level support, such as avirtual SVGA driver 153 a. Each VMX process and VMM pair cooperate toprovide the effective (and isolated) virtualization executionenvironment for each VM to run. In general operation, the virtualizationlogic 120 receives requests from the virtualized device driversimplemented by the VMMs and VMX processes, translates (or otherwisetransfers, forwards, sends, or communicates) these requests tocorresponding requests to real device drivers 130 or 122 thatcommunicate with real devices resident in the host hardware platform 101(such as frame buffer 104, NIC 105, etc.).

The various terms, layers, categorizations, components used to describethe virtualization server computing system 100 of FIG. 1 may be referredto differently without departing from their functionality or the spiritof this disclosure. Also, one or more of the components may not bepresent in any specific implementation. For example, the virtualcomponents shown as part of virtualization logic 120 that are notincluded in each VMM 142 a-142 c (for example, one or more of components130-134, 121-122, or the like) may be considered in other embodiments tobe part of the VMMs 142 a-142 c. In addition, in some embodiments, noVMX process is used and the MKS thread capabilities, including the UIremoting, and virtual graphics adaptor support are integrated insteadinto the VMMs 142 a-142 c or into other parts of the virtualizationlogic 120. Also, in some embodiments the VMMs 142 a-142 c may beconsidered to be separate from or part of the VM 102. Embodiments of theUIROS may be practiced in other virtualized computing environments suchas hosted virtual machine systems, where the virtualization logic 120 isimplemented on top of an operating system executing on host hardwareplatform 101 instead of directly on the host hardware.

Furthermore, in some embodiments, some or all of the components of thevirtualization server computing system 100 may be implemented orprovided in other manners, such as at least partially in firmware and/orhardware, including, but not limited to one or more application-specificintegrated circuits (ASICs), standard integrated circuits, controllersexecuting appropriate instructions, and including microcontrollersand/or embedded controllers, field-programmable gate arrays (FPGAs),complex programmable logic devices (CPLDs), and the like. Some or all ofthe components and/or data structures may also be stored as contents(e.g., as executable or other machine-readable software instructions orstructured data) on a computer-readable medium (e.g., a hard disk;memory; network; other computer-readable medium; or other portable mediaarticle to be read by an appropriate drive or via an appropriateconnection, such as a DVD or flash memory device) such as computerreadable medium 110 to enable the computer-readable medium to execute orotherwise use or provide the contents to perform at least some of thedescribed techniques.

FIG. 2 is an example block diagram of components of an example UserInterface Remoting and Optimization System. In one embodiment, the UserInterface Remoting and Optimization System (UIROS) comprises one or morefunctional components/modules that work together to send user interfaceupdates to a client device using video compression and streamingtechniques. These components may be implemented in software, hardware,firmware, or a combination. In FIG. 2, a UIROS comprises server sidesupport that resides on one or more host or server computing systems 201and client side support that resides on one or more client devices 220.

In an example embodiment, the server side support includes a UI remotingserver 204, a video encoder 205, and rendering support 206. In someembodiments, these components execute as part of the VM Support 203, forexample, as part of a process (e.g., the VMX process in VMware'svirtualization environment) that executes on virtualization logic 202,which is hosted by host/server computing system 201. For example, thesecomponents 204-206 may execute as part of an MKS (mouse, keyboard,screen handling) thread 152 a, which executes as part of VMX processes151 a-151 c as described with reference to FIG. 1. In other embodiments,these components may be implemented in other parts of the virtualizationenvironment such as part of each VMM (virtual machine monitor, e.g.,VMMs 142 a-142 c) or as other parts of virtualization logic 202. Therendering support 206 is responsible for receiving the virtual graphicsdevice commands from guest 210 (guest applications 213 executed from thedesktop 212 using the guest operating system 211) and carrying them outthrough the graphics stack (shown in FIG. 1 as graphics API 121 andgraphics drivers 122) to the graphics hardware associated with the host201, such as frame buffer 104. The video encoder 205 is responsible forencoding the user interface updates as will be described in detailfurther herein, when invoked by the user interface remoting (UIR) server204. The UIR server 204 can transmit user interface updates to aconnected client device 220 through a TCP/IP connection 207 in the VMsupport 203 (virtualization logic 202, or host 201) or through a TCP/IPconnection 217 in the guest operating system (guest OS) 211 for someclient device interfaces that may be optimized to receive UI remotingfrom the guest OS 211 instead of from the virtualization support 201,202, or 203.

The client device 220 receives the UI display updates through its TCP/IPconnection 227 and user interface remoting (UIR) client 224. The UIdisplay updates may be initially received through web browser codeconnected through the TCP/IP connection, which passes the information(e.g., redirects, forwards, communicates, or sends it) to the UIR client224 and/or the video decoder 225. For example, in some embodiments, theUI display updates are handled by the web browser code's implementationof a <video> tag, for example, as available under the HTML 5 standard,to send the updates to the video decoder 225. In other embodiments, forexample those that use an intermediary, such as UIR client 224, the webbrowser code may pass the video bitstream to the UIR client 224. The UIRclient 224 may be implemented, for example, as Javascript downloaded tothe client device upon connection to the VM (or at other times), tometer how the video is processed as opposed to using whateverimplementation the browser code supports for the <video> tag. The videodecoder 225 is then invoked (e.g., by the web browser code or by the UIRclient 224) to reconstruct the bitstream (e.g., from a previous cachedframe on a move, scroll, expose region, or similar operation) and sends(e.g., forwards, transmits, communicates, etc.) the data stream to berendered on display 226.

In example embodiments, the components of the server side support andclient side support of a UIROS are implemented using standardprogramming techniques. In general, a range of programming languagesknown in the art may be employed for implementing such exampleembodiments, including using object-oriented, functional, scripting, anddeclarative languages. In addition, in other embodiments thefunctionality of the different components may be distributed amongstthemselves in different ways or distributed in other ways, yet stillachieve the functions of a UIROS.

As mentioned, in some embodiments, the UIR server 204, UIR client 224,video encoder 205, and video decoder 225 utilize video streaming andcompression techniques defined by the H.264 standard, although otherstreaming and compression techniques may be substituted. FIG. 3 is ablock diagram of an overview of example video streaming using an examplegeneric video streaming protocol. In this example, a video source 300 isencoded and transmitted to generate video output 315 using techniquesthat minimize updates based upon recognizing that previous data can beused to generate a current frame. In particular, the video source 300 isexamined by prediction logic 301 (e.g., code, component, block, program,task, etc.) using a process known as motion estimation to determine towhich other data the current block to be rendered corresponds. In someembodiments, other data in the current frame is used to predict the datain the current block (“intraframe” prediction). In other embodiments,data from a block in a prior transmitted frame is used to predict thedata in the current block (“interframe” prediction). For video, avariety of motion estimation algorithms may be used including exhaustivesearching. When using the H.264 standard, blocks of different sizes maybe used, including 16×16 pixel blocks down to 4×4 pixel blocks. Once theprediction block is selected, and it is determined how the block is tobe relocated to result in the current (desired) block to be rendered,the data is transformed by transform component 302 (e.g., module, logic,block, code, program, task, etc.) according to techniques of thestreaming and compression protocol and then encoded (and typicallycompressed) into a bitstream by encode component 303. The encodedbitstream is then transmitted or stored 310 in an appropriate syntax tobe delivered to a target (e.g., client) device based upon the videoformat. When received on the client device, the encoded (and typicallycompressed) bitstream is then decoded by decode component 311 and thenan inverse transform is conducted on the information in the bitstreamusing transform component 312 to resurrect information for building theblock to be rendered. The transform data along with other data from thebitstream (for example, a pointer, reference, etc. to a predictionblock) is then used by the reconstruct component 313 to produce videooutput 315 for rendering.

When using the H.264 standard, the UIROS determines a prediction block(called a “prediction macroblock”) and calculates a motion vectorencoding how that block is to be moved to efficiently represent manyuser interface activities addressed in UI remoting. FIG. 4 depicts anoverview of an example video encoding technique performed according tothe H.264 video streaming protocol. The example depicted utilizesinterframe prediction (using a previously encoded and transmitted frame)to perform encoding. That is, a prior frame I 401 is used to find amacroblock 402 that is similar to a macroblock 412 in the current frameI+1 410 to be encoded and transmitted. Prediction macroblock 402 in apast transmitted frame 401 is depicted as containing the exact samepixel data that is to be presented in the macroblock 412 which is beingencoded. The motion vector 413 in the current frame 410 being encodedcaptures the movement of that macroblock from its prior position (411)to its destination position (412). According to the H.264 standard, aresidual macroblock 414 is computed that contains differences betweenthe prediction macroblock 402 and the target macroblock 412 beingencoded. Due to the nature of UI encoding (bit-blit operations), theresidual macroblock 414 is typically null/empty because the informationis not changed, just moved. The H.264 standard specifies how theprediction macroblock 402 is encoded, along with motion vector 413, sothat the decoder can reuse an already received block of cached pixeldata (411), move it to the position indicated by motion vector 413, andapply any information contained in the residual macroblock 414 to renderthe resultant macroblock 412. The H.264 standard specifies how all ofthis information is encoded into a compressed bitstream that is compactand efficient. More information on the H.264 standard is described inkin Richardson, White Paper: A Technical Introduction to H.264/AVC,VCcodex, 2002-2011, and in lain Richardson, White Paper: An overview ofH.264 Advanced Video Coding, Vcodex/OnceCodec, 2007-2011 incorporatedherein by reference in their entireties.

When used with UI remoting, the UIR server (e.g., UIR server 204 of FIG.2), incorporates knowledge of user interface activities to performmotion estimation—to determine an appropriate prediction macroblock foreach block being encoded and transmitted to the client device. Forexample, when a user scrolls a document or moves something on thedesktop, for example a window, the UIR server can compute what pixelsremain the same and what is the new content (additional pixels) for theclient device to render. Using H.264 (or any other similar videoencoding) the UIR server can send to the client device indications ofpredictive macroblocks and their motion vectors to correspond to theportions of the user interface that are still to be rendered but in adifferent location. The UIR server then only needs to encode as new datathe pixels that are now to appear on the display screen (be rendered bythe client device).

FIG. 5 depicts an example of pixel movement according to an exampleoperation of a user interface. In this case, the example operation is ascrolling operation. This example depicts a document in Frame I 501being manipulated on a virtualization server. The content of thedocument presently being displayed (before the user interface operation)is shown in Frame I 501. These are the pixels that have already beenremoted to a client device. The content that is available in the text tobe displayed, but not yet displayed, is shown as text 504. When a userperforms a scroll down operation (the text scrolls up), the content withdotted box 503 in Frame I 501 is already cached on the client device andtherefore can be encoded as prediction macroblocks and motion vectors505 that will reflect the content (indicated as within dotted frame 513)in a new position in Frame I+1 510. The new content is shown withindotted box 515. The pixels in box 515 are the pixels that need to beencoded and transmitted to the client device to complete rendering ofthe scrolling operation. Thus, when the UIR server uses video encodingto encode the outcome of the scroll operation, the UIR server will useprediction macroblocks from Frame I 501 and motion vector 505 tocommunicate to the client to reuse the pixel data shown in dotted box513 moved to a new position and to render new (encoded and transmitted)content 515.

Although the examples described herein often refer to remoting a userinterface desktop and such actions as scrolling, moving, grabbing,pinching, and exposing regions or pixels, the techniques describedherein can also be used to render any type of display presentation. Inaddition, the concepts and techniques described are applicable to othervideo encoding techniques, including other types of video encodingtechnologies and browsers that support the same. Also, although certainterms are used primarily herein, other terms could be usedinterchangeably to yield equivalent embodiments and examples. Inaddition, terms may have alternate spellings which may or may not beexplicitly mentioned, and all such variations of terms are intended tobe included.

Example embodiments described herein provide applications, tools, datastructures and other support to implement a User Interface Remoting andOptimization System to be used to remote UIs using video encodingtechnology. In the following description, numerous specific details areset forth, such as data formats and code logic sequences, etc., in orderto provide a thorough understanding of the described techniques. Theembodiments described also can be practiced without some of the specificdetails described herein, or with other specific details, such aschanges with respect to the ordering of the logic, different logic, etc.Thus, the scope of the techniques and/or functions described are notlimited by the particular order, selection, or decomposition of aspectsdescribed with reference to any particular routine, module, component,and the like. For example, given the time criticality of the actionsinvolved in UI remoting, the decomposition into multiple sequences asdepicted in FIGS. 6-9 may not likely to be reflected in a liveimplementation, but is so depicted for ease of description.

FIG. 6 is a flow diagram of example server logic for enhanced userinterface remoting according to an example embodiment. Logic 600 may beexecuted, for example, within MKS thread 152 a of the UIR server 204 inFIG. 2 of virtualization logic 120 in FIG. 1. This logic illustrates theoverall flow for remoting a UI (such as a user desktop) to a clientdevice.

In block 601, the logic analyzes the user interface subject to remotingfor reusable pixel information in the portion of the user interface thathas changed. This analysis is described in detail with respect to FIG.7. In overview, the UIR server determines whether the UI command is onethat results in a bit-blit operation (a block transfer of pixel data) ofalready transferred pixel data to a possibly modified location. If so,then the operation is likely one that can be represented by an optimizedP-frame using the video encoding techniques described. If not, then theoperation may encode the changed regions by sending one or moremacroblocks with new pixel data. The reusable pixel techniques describedwith respect to FIG. 7 are performed relative to the portion of the UIthat has changed (the “dirty” region or changed portion of the display).The portions of the UI that have not changed may be encoded using videoencoding techniques by reusing macroblocks from prior frames, or in someembodiments, by indicating unchanged regions in the video stream (e.g.,by using a construct that indicates what macroblocks can be skippedover).

In block 602, the logic calibrates generation of the video stream to beproduced. The UIR server tries to generate frames to send to the clientdevice just at the right frequency and point in time so that they makeit to the client device in a timely manner and are not “stale” in thepipe in the case of overloaded network queues. To do this, the UIRserver logic calibrates generation of the video stream (before andpotentially) while it is being delivered to the client device.Calibration may occur, for example, as a result of obtaining feedbackinformation from a client device (over, for example, a backchannel)regarding network conditions and client load. Feedback may includenetwork latency and bandwidth information as well as client overloadconditions. The UIR server may use this information along with otherinformation about the client device, such as the device type, model,etc. to calibrate its synthetic and real time generation of the videostream. When the UIR server determines that the network is overloaded(e.g., the transmit buffer is getting overloaded), the UIR server maycompress harder (e.g., more compression of the encodings) or slow downthe generation of the video frames. In some embodiments, depending uponthe encoding/decoding, when bandwidth is limited, the UIR may utilizeresidual frames to deliver progressive enhancements that the client mayrender accumulatively, thus leveraging the additive construct ofresidual frames. When, on the other hand, the UIR server determines thatthe client device (not the network) is overloaded, the UIR server maytransmit fewer frames. Similarly, when the UIR server determines thatthere is abundant bandwidth to deliver data to the client device, theUIR server may compress lighter to reduce the CPU processing on the hostserver needed to provide compression of pixel data. Other calibrationsmay be similarly incorporated. The logic of block 601 may be performedat a variety of times, but is typically performed prior to transmittingthe video encoding to the client device in block 609.

In block 603, if the analysis of block 601 returned an indication thatsuch a bit-blit operation of previously transferred data is occurring(e.g., due to a move, scroll, expose, or other UI operation), then thelogic continues in block 604, otherwise the logic continues in block605.

In block 604, the logic computes the encoding according to the videoencoding protocol, for example, H.264 described above and continues inblock 609. An example of the computation of the video encoding isdescribed further in FIG. 8 and accounts for reusable pixel informationas well as the data that needs to be sent anew.

In block 605, when the analysis of block 601 returned an indication thatan operation related to previously transferred data is not occurring,then the logic determines whether an operation to play video has beenindicated. If so the, logic continues in block 606, otherwise continuesin block 608.

In block 606, the logic computes an encoding for remoting the indicatedvideo playback, and then continues in block 609. This operation mayoccur, for example, when a user is using a video on demand applicationsuch as YouTube or a video conferencing application like Skype. In thiscase, the logic attempts to leverage the source video stream inconstructing the remoted video stream. Such an approach is significantlymore CPU efficient than decoding the video stream into pixels, only tore-encode it onto another video stream as part of the UI. If for somereason the client device is not capable of decoding the original videostream, or the available bandwidth to the client device is insufficientto receive the source video stream (or for other reasons), then thelogic transcodes the source stream into a proper form to remote. In someembodiments, the motion vector information and macroblocks from theoriginal video stream can be reused at least in part to speed up thetranscoding process so that at least a portion of the original videoencoding can be leveraged. For example, the coordinate system for thearea of the UI reserved for the video stream may be different from thecoordinate system used for the UI remoting as a whole; however thecalculations of the new motion vectors may still be informed by thecalculations of the motion vectors of the video stream

In block 608, if the logic has determined that some other UI operationhas occurred that requires remoting but for which no motion estimationalgorithm is used or is not effective, then the logic continues toencode pixel data using standard procedures, for example, usingmacroblocks with new data when data has changed, or to reuse macroblocksfrom prior frames or to skip macroblocks when data is unchanged, asappropriate, and continues in block 609.

In block 609, the logic transmits the encoded data (in an encodedbitstream as dictated by the video protocol employed) to the UI remotingcode (e.g., UIR client 224 in client device 220 in FIG. 2). In someembodiments, this is accomplished using a network connection (e.g.,TCP/IP connection 207 in the virtualization logic or host) to the webbrowser (not shown) running on the client device, which has previouslyestablished an authenticated connection to the server's TCP/IPconnection. The web browser can act as a thin veneer and passes thevideo stream straight to the video decoder (often hardware assisted),for example, video decoder 225, which then renders the data on thedisplay of the client device. In some embodiments, code is downloaded tothe client device, for example, as Javascript, to meter the feeding ofthe video stream to the <video> tag processing by the browser so as tocontrol the speed of processing of the video stream. In otherembodiments, the TCP/IP connection to the web browser operates throughthe guest OS, such as TCP/IP connection 217 in guest OS 211. In stillother embodiments, the encoded data is sent over TCP/IP directly to adriver (not shown) for the video decoder native to the client device.Other embodiments may employ other networking protocols other thanTCP/IP. FIG. 2 depicts these different connections as “UIR protocol.”

FIG. 7 is a flow diagram of example logic for analyzing a user interfacefor determining reusable pixel data. Logic 700 may be executed, forexample, within MKS thread 152 of the UIR server 204 in FIG. 2 ofvirtualization logic 120 shown in FIG. 1. Logic 700 may be invoked, forexample, from block 601 in FIG. 6. This logic determines whether thenext changed region to be encoded may be optimized to reuse existingmacroblock information, taking advantage of UI frame coherence, or needsto be sent as a key frame (a new frame), for example, when the useropens a screen wide application. One or more of the techniques used inthis logic sequence may be implemented in any embodiment. For example,in the embodiment shown, each technique is examined to determinereusability (in a series). In other techniques, one or more techniquesare examined in the alternative. In one example embodiment, a genericvideo encoder is modified to implement one or more of these techniquesin its motion estimation portion of its logic.

In block 702, the logic inspects a frame buffer to detect whether ascrolling or move type event has occurred (looking for a scrolling ormove “fingerprint”) and to determine the reusable pixel informationtherefrom. In some embodiments this is done by detecting a bit-blit bycomparing the frame buffer content to a previous frame buffer contentupon receiving mouse movement, arrow keys, scroll dial events, and otheruser input that would indicate a “move” of previously visible pixels. Ifa bit-blit has been detected, appropriate prediction macroblocks andtheir corresponding motion vectors are generated. Here, the frame bufferis typically the virtual frame buffer utilized by the VM thatcorresponds to the executing application. Other constructs such as“blitmap” structures (not shown), such as those implemented by a virtualvideo adapter driver, may be used to support determining whether thereare regions in the frame buffer that have changed during some timeinterval or as a result of certain operations. (A description of thecreation and (other) use of a blitmap structure is provided in Byford etal., US 2010/0271379, entitled “Method and System for Copying aFramebuffer for Transmission to a Remote Display,” published on Oct. 28,2010.) Also, in some systems, a prior state of the frame buffer (in partor in whole) may be cached for comparison to determine reusable pixelinformation. The logic then continues in block 704 to determineadditional possible optimizations keeping track of the ones it hasalready generated.

In block 704, the logic inspects the guest's graphics command stream todetect move, scroll, expose or other type of relevant UI events (by, forexample, pattern matching particular commands). If so, appropriateprediction macroblocks and their corresponding motion vectors aregenerated, for example, using information from the previously encodedframes maintained by, for example, the video encoder 205 of FIG. 2. Thelogic then continues in block 706 to determine additional possibleoptimizations keeping track of the ones it has already generated.

In block 706, the logic determines whether it has received directives(through, for example, notification via an API) of any UI semantics. Insome embodiments, the logic supports an API that enables applications(such as a new class of application, for example a “pixel oriented cloudapplication” as opposed to an HTML based application) to be structuredand coded to take advantage of UI coherence information and knowledgethat its UI may be remoted, for example, in a virtualizationenvironment. An application having awareness of UI remoting (directly orindirectly, for example, through some type of application framework thathides some or all of the UI remoting support) may opportunisticallycache pixel information, for example, a next new block of content ofscroll pixels, or a next photo in a slide show, before it is to berendered on a display screen. For example, applications (or othercomponents that aid in UI remoting) may leverage H.264 decoder framecaching to pre-load pixel information to provide a type of “lookahead”to decrease bandwidth usage and UI latency on the client device.

If the logic determines it has received notification (e.g., directives,commands, determination, instructions, etc.) of UI semantics, then thelogic continues in block 708 otherwise, continues to block 710. In block708, the logic determines whether the received notification pertains toone of these commands and if so continues in block 709, otherwisecontinues to block 710. In block 709, the logic generates appropriatereusable pixel and/or other information as described below, and thencontinues to block 710.

In block 710, the logic returns the generated pixel informationincluding the prediction macroblocks and their corresponding motionvectors. If no optimizations are possible, then new pixel information(one or more macroblocks) are returned.

As mentioned, in block 706, the logic may receive commands from theguest application that indicate (e.g., through the use of an API) thatthe application is using cached pixel information, through the use oflookahead techniques (using reference frames) to implement certain UIoperations. Common UI patterns such as scrolling, stretch/pinch, windowgrab and move, panning through the UI canvas, etc. can be implemented byhaving the application provide guidance for the encoding process tostore lookahead pixel information as reference frames in preparation forusing them in the client to lower bandwidth use and/or latency. Forexample, in response to application directives (implemented for exampleusing an API), pixels can be calculated (e.g., composited, transformed,scaled, rotated, rendered, and the like) and then encoded using thevideo encoding techniques described herein and sent to the client deviceto be stored as reference frames. Latency can then be further reduced byperforming such calculations ahead of their use.

The following common UI patterns provide some exemplary examples where alookahead (pixel caching) mechanism may be useful:

-   -   Scrolling (e.g., through a linear document): This UI pattern may        occur, for example, while a user views any type of document. In        addition to transmitting the currently viewed section of the        document, adjacent sections may be preloaded predicatively. As        the user scrolls in either direction, a large portion of the        displayed document is reused (by specifying reference frames to        already transmitted pixel data), while only a band of new pixels        (preloaded) is displayed.    -   Pinch and Stretch: This UI pattern may occur, for example, in        photo or map-based applications. The application (through the        API) may generate and use a mipmap representation (e.g., a        collection of representative images corresponding to different        sizes of the map) while causing the appropriate level of        resolution to be preloaded. Depending upon the user input, the        application can predictably preload the next level of resolution        and switch to the new resolution as appropriate.    -   Window Grab and Move: This UI pattern may occur, for example,        when a user grabs a window (e.g., depressing a pointer device on        the window) and drags it to a new location (e.g., without        releasing the depressed pointer device). The preloaded window        may be moved (using motion vectors) without resending the pixel        data to the client. Likewise, uncovered pixel regions could be        preloaded in reference frames and displayed by simply using        predictive macroblocks that point to these reference frames.    -   Slideshow: This UI pattern may occur, for example, when a user        is viewing a slideshow of a photo collection. An application        that allows a user to view this slideshow may scale a given        photo based upon the resolution of the target client device        before rendering. As the user is viewing a photo in the        slideshow mode, the next photo may get preloaded (and        transmitted) by the server to be stored as a reference frame to        enable a smooth transition from photo to photo. In addition, if        a user pauses the slideshow, the application (through the API)        may then preload different resolutions in anticipation of user        manipulation. When a user stretches the photo, a new higher        resolution instance (already preloaded and encoded as a        reference frame) is referenced by the video stream.    -   Panning through a UI Canvas: This UI pattern may occur, for        example, when the client is a small footprint device such as a        mobile device. The user may want to navigate with ease the user        interface (or whatever is being presented) in some direction,        for example up, down, left, and right. The application can        “preload” frames that are adjacent to the one currently being        view by the user in order to provide a smoother shift to the        adjacent content.

Generally, these techniques can be used to mitigate network limitationsin order to provide smooth user interactivity. Other UI patterns may besimilarly implemented.

FIG. 8 is a flow diagram of example logic for determining and generatingan encoding of user interface data using a video streaming protocolaccording to an example embodiment. Logic 800 may be executed, forexample, within MKS thread 152 of the UIR server 204 in FIG. 2 ofvirtualization logic 120. Logic 800 may be invoked, for example, fromblock 604 in FIG. 6 when at least some pixel reusability is possible.This logic determines the encodings according to the process describedwith respect to FIG. 3 and is specific to the video encoding protocolemployed.

In block 801, the logic determines one or more prediction macroblocksfrom a prior transmitted frame that may be reused, as indicated from theanalysis of the logic of FIG. 7 and as described in FIG. 4. This is donefor whatever portions of the display are being remoted since, in someexamples, the entire display screen may or may not be remoted. Theresults of the motion estimation of FIG. 7 will indicate what rectanglesare reusable. For example, when a document is being scrolled, there aresome portions of the changed regions of the display (e.g., macroblocks)that are determined to be reusable, and other portions of the changedregions that may need to be sent as new data (e.g., new macroblocks).

In block 803, the logic determines the one or more corresponding motionvectors applicable to the prediction macroblocks, as indicated from theanalysis of the logic of FIG. 7 and as described in FIG. 4.

In block 805, the logic encodes the remaining rectangles of the changedregions as new macroblocks, since no reusability has been determined forthis pixel information.

In block 806, the logic encodes the unchanged regions of the portion ofthe display being remoted. As described above, this may be performed byreferencing existing macroblocks previously transmitted, by indicatingwhich macroblocks may be “skipped,” or by other mechanisms.

In block 807, the logic compresses and encodes the one or moreprediction macroblocks, their corresponding motion vectors, and theremaining macroblocks and information and transforms and compresses thisinformation into a bitstream according to the requirements of the videoencoding protocol. The encoded bitstream is returned at block 808.

FIG. 9 is a flow diagram of example client logic for decoding andrendering a received video bitstream according to an example embodiment.Logic 900 may be executed, for example, by the video decoder 225 in FIG.2 executing, for example, within the client device 220. Logic 900receives and processes the encoded bitstream that results from FIGS. 6and 8 according to the process described with reference to FIG. 3.

In block 901, the logic receives an encoded bitstream (e.g., from theUIR client 224 via a web browser over a TCP/IP connection).

In block 903, the logic decompresses and decodes the bitstream accordingto the video stream protocol to determine the one or more predictionmacroblocks and their associated motion vectors and residualmacroblocks. This may include performing an inverse transform torecreate the residual macroblock information.

In block 904, the logic computes the pixels to render by locating thecached pixel information based upon the indicated predictionmacroblocks, moves the pixels according to their respective motionvectors, and adds any difference information encoded in the residualmacroblock information. Note that in the typical case, bit-blits do notresult in changed information so that the residual macroblockinformation is non-existent (or zero). In some embodiments, for examplethose in which bandwidth is scarce, the residual macroblocks are usedfor progressive refinements. In some cases, I-Frames (key frames) areencoded, compressed, and transmitted, and thereby rendered as is.

In block 907 the computed pixel information is rendered on a displaydevice associated with the client device.

The logic then ends.

All of the above U.S. patents, U.S. patent application publications,U.S. patent applications, foreign patents, foreign patent applicationsand non-patent publications referred to in this specification and/orlisted in the Application Data Sheet, are incorporated herein byreference, in its entirety.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the present disclosure. For example, the methods, techniques,and systems for performing video encoding for UI remoting discussedherein are applicable to other architectures other than an x86architecture. Also, the methods and systems discussed herein areapplicable to differing protocols, communication media (optical,wireless, cable, etc.) and devices (such as wireless handsets,electronic organizers, personal digital assistants, portable emailmachines, tablets, notebooks, smart televisions, game machines, pagers,navigation devices such as GPS receivers, etc.).

1. A method in a server computing system for efficiently remoting a userinterface on a client device by reusing pixel information, comprising:receiving an indication from an application that causes one or more setsof pixels to be stored in lookahead reference frames; determining from auser interface aspect of the application, prior to updating the remoteduser interface by causing an updated portion of the user interface to berendered on a display screen associated with the client device, whetherat least one of the sets of pixels stored in a lookahead reference framecan be reused to accomplish the updating; when at least one of the setof pixels stored as a lookahead reference frame cannot be reused toaccomplish the updating, using a video encoding protocol to generate andencode a video encoded representation of the updated portion of the userinterface using one or more macroblocks with new pixel information; whenat least one of the set of pixels stored as a lookahead reference framecan be reused to accomplish the updating, using a video encodingprotocol to generate and encode a video encoded representation thatrefers to the determined at least one of the sets of pixels stored as alookahead reference frame, instead of sending new pixel data; andsending the encoded video encoded representation to the client device ina manner that causes a video decoder on the client device to render theupdated portion.
 2. The method of claim 1 wherein the video encodingprotocol is at least one of an H.264 protocol or VP8 protocol.
 3. Themethod of claim 1 wherein the video encoded representation of theupdated portion reuses a set of pixels stored as a lookahead frame byincluding a reference to a reference macro block and a motion vectorinstead of encoding and transmitting new pixel data.
 4. The method ofclaim 1 wherein the receiving an indication from the application furthercomprises receiving an indication via an Application ProgrammingInterface (an “API”).
 5. The method of claim 1 wherein the analyzing theuser interface aspect of the application is performed as part of animplementation of an API.
 6. The method of claim 1 where the one or moresets of pixels caused to be stored as lookahead reference frames aretransmitted to the client device prior to generating and encoding avideo encoded representation that refers to the one or more sets ofpixels.
 7. The method of claim 1 wherein the determining from the userinterface aspect of the application whether at least one of the sets ofpixels stored as the lookahead reference frame can be reused toaccomplish the updating comprises receiving indications from theapplication of one or more of: scrolling, pinch and/or stretching, threedimensional effects, window movement, and/or window exposure.
 8. Themethod of claim 1 wherein the receiving an indication from anapplication that causes one or more sets of pixels to be stored inlookahead reference frames further comprises: on the server computingsystem, before sending the encoded video encoded representation to theclient device, generating pixels to be stored in one or more of thelookahead reference frames by calculating the pixels; and storing thecalculated pixels in the one or more lookahead reference frames.
 9. Themethod of claim 8 wherein calculating the pixels comprises performing amathematical transformation of the pixels.
 10. The method of claim 9wherein the mathematical transformation of the pixels comprises at leastone of compositing or rendering the pixels.
 11. The method of claim 9wherein the wherein the mathematical transformation of the pixelscomprises at least one of scaling or rotating.
 12. The method of claim 1wherein, when at least one of the set of pixels can be reused toaccomplish the updating, the at least one of the set of pixelsrepresents an adjacent portion of a document a next image in a sequence,a covered portion of a UI, and/or an adjacent region of a UI canvas. 13.The method of claim 1 wherein, when at least one of the set of pixelscan be reused to accomplish the updating, the at least one of the set ofpixels represents an image of a resolution or size different than thatof the same image currently being rendered by the client device.
 14. Themethod of claim 1 wherein, when at least one of the set of pixels can bereused to accomplish the updating, a plurality of the sets of pixels arestored as lookahead reference frames.
 15. The method of claim 1 whereinthe server computing system is a virtualization server computing systemand the updated portion of the user interface is a portion of a desktopcorresponding to a connection to a virtual machine.
 16. The method ofclaim 1 wherein the server computing system is a physical computingsystem.
 17. The method of claim 1 wherein the method is performed by avirtual machine process and/or a virtual machine monitor of avirtualization server computing system.
 18. The method of claim 1wherein the client device is at least one of a personal computer, awireless or wirelessly connected device, a phone, a television, and/or atablet.
 19. The method of claim 1 wherein the application is acloud-based application not tied to a particular server.
 20. Acomputer-readable medium stored in a server containing content forremoting a user interface on a client device by reusing pixelinformation, by performing a method comprising: causing one or more setsof pixels to be stored in lookahead reference frames; determining from auser interface aspect of the application, prior to updating the remoteduser interface by causing an updated portion of the user interface to berendered on a display screen associated with the client device, whetherat least one of the sets of pixels stored in a lookahead reference framecan be reused to accomplish the updating; when at least one of the setof pixels stored as a lookahead reference frame cannot be reused toaccomplish the updating, using a video encoding protocol to generate andencode a video encoded representation of the updated portion of the userinterface using one or more macroblocks with new pixel information; whenat least one of the set of pixels stored as a lookahead reference framecan be reused to accomplish the updating, using a video encodingprotocol to generate and encode a video encoded representation thatrefers to the determined at least one of the sets of pixels stored as alookahead reference frame, instead of sending new pixel data; andsending the encoded video encoded representation to the client device ina manner that causes a video decoder on the client device to render theupdated portion.
 21. The computer-readable medium of claim 20 whereinthe medium is a memory of a computing system and the content is computerinstructions stored in the memory.
 22. The computer-readable medium ofclaim 20 wherein the method is performed as part of an APIimplementation.
 23. The computer-readable medium of 20 wherein themethod is performed as part of an application framework implementation.24. The computer-readable medium of 20 wherein the user interface aspectcomprises user interface commands received from an application executedby the server computing system.
 25. The computer-readable medium ofclaim 20 wherein determining from the user interface aspect whether atleast one of the sets of pixels stored as the lookahead reference framecan be reused to accomplish the updating comprises recognizing at leastone of a scrolling operation, a window move operation, a window exposeoperation, or a pan operation across a UI canvas.
 26. Thecomputer-readable medium of claim 20 wherein the video encoding protocolis at least one of an H.264 protocol or an VP8 protocol.
 27. Thecomputer-readable medium of claim 20 wherein the video encodedrepresentation of the updated portion of the user interface comprises areference to a reference macro block and a motion vector.
 28. Thecomputer-readable medium of claim 20 wherein the computer-readablemedium is stored as part of a virtualization environment and theapplication is executing as part of a virtual machine environment.